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Background 


Environment 

■  Planned  investments  in  DOD 
weapons  systems 

*  $700  billion  in  2001 

*  $1.4  trillion  in  2006 

■  Cost  of  development  exceeding 
estimates  by  30-40% 

■  Fewer  quantities  purchased  than 
anticipated 

■  Longer  development  schedules 
needed 
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Background 


Our  Best  Practices  Work 


Leading  companies 
we’ve  visited 

Companies’  best  practices 

■  Motorola 

■  Caterpillar 

■  Disciplined  software  and 
management  processes 

■  Prototype  testing  to  improve 
reliability 

■  Program  management  practices  to 
encourage  collaboration 

■  Requirements  that  specify 
reliability  and  total  ownership  costs 

■  Collection  of  metrics  data  to 

■  Toyota 

■  FedEx 

■  NCRTeradata 

improve  software  reliability 
■  Technology  readiness  levels  and 

■  Boeing 

design  maturity 

■  Hughes  Space  and 

■  Statistical  control  over  production 

Communications 

processes 
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Background 


Best  Practices’  Knowledge  Points 


Knowledge  is  gained  at  key  development  points 
during  product  development 

Knowledge  point  1 

■  Match  is  made  between  customer’s 
wants  and  resources  (i.e.  technology, 
design,  time,  and  funding) 

★  Critical  technologies  should  have 
reached  Technology  Readiness 
Level  7  or  higher 

Knowledge  point  2 

■  Product’s  design  demonstrates  ability 
to  meet  performance  requirements 

★  High  percent  of  design  drawings 
released  to  manufacturing 

Knowledge  point  3 

■  Product  can  be  manufactured  within 
cost,  schedule,  and  quality  targets 
and  is  reliable 

★  High  percent  of  production 
processes  under  statistical  process 
controls  and  achieving  Cpk  of  1 .33 
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Background  Our  Recent  Report 

■  Best  Practices:  Better 
Support  of  Weapon  System 
Program  Managers  Needed 
to  Improve  Outcomes 
(GAO-06-1 10)  and 
Survey  of  Weapon  System 
Program  Managers  (GAO-06-1 12SP) 

★  How  does  DOD  support  its  program  managers,  and 
how  does  it  hold  them  accountable  for  program 
outcomes? 

★  How  do  leading  companies  support  their  program 
managers,  and  hold  them  accountable  for  program 
outcomes? 

★  What  can  DOD  do  to  better  position  its  program 
managers  for  successful  outcomes? 
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Background 


Our  Report’s  Methodology 

■  Case  studies  of  leading  companies 

★  Motorola,  Siemens  Medical  Group, 
Toyota  Avalon,  Wells  Fargo,  and 
Moulson-Coors  Inc. 

■  Literature  reviews 

■  Focus  groups 

★  5  locations,  28  program  managers 

■  Survey 

★  Category  1  and  2  program  managers 

■  Interviews 
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Best 

Practices 


Success  Factors  for 
Program  Managers 

■  Critical  support  and  accountability 
factors 

■  Investment  strategy 

■  Business  case 

■  Knowledge-based  product 
development 

■  Accountability/rewards 

■  Other  success  factors 
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Best 

Practices 


Critical  Support  and 
Accountability  Factors 


X 

n 


Senior 

leadership 


Develop  long-term  vision 
and  investment  strategy 


Support  program 
manager 


Hold 

accountable 


Train,  mentor 

program  manager 


Empower  program 
manager 


Develop  business  case, 
assign  to  program 
manager 


CONCEPT  DEVELOPMENT 


PROGRAM 

START 


DESIGN 


PRODUCTION 


Gap  between  resources 
and  requirements 
is  closed 


Knowledge-based  process  is  followed; 
information  on  cost,  schedule,  design,  and 
production  maturity  is  demanded  throughout 


A 


Program  manager 
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Best 

Practices 


Investment  Strategy 

■  Consistent  with  company  strategic 
vision 

*  Corporate  leadership  accountable 

■  Forecast  market  needs 

*  Economic  trends,  market  position, 
technologies 

■  Long-term  and  short-term  planning 

*  Project  selection  and  prioritization 
make  trade-offs  that  fit  within 
corporate  goals 
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Best 

Practices 


Business  Case 

■  Match  resources  and  requirements 
using  systems  engineering 

+  evolutionary  product  development 

★  achievable  requirements 

•  short  cycle  times 

★  estimate  time,  money,  technologies, 
people 

★  program  manager  assigned  for 
development  duration 

■  Goal — close  gap  between  customer 
wants  and  available  resources  before 
committing  to  development 
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Best 

Practices 


Knowledge-Based  Product 
Development 

■  Gated  process  that  builds  knowledge  over 
time 

★  Program  manager  accountable  for  execution 

★  Frequent  reviews,  decision  point  meetings  with 
program  manager  and  senior  level  managers 

★  Management  decisions  based  on  data 
submitted  by  program  manager 

♦  metrics,  such  as  earned  value,  percentage  of 
design  drawings  completed,  process  controls 
under  statistical  control,  tests  completed 

★  Development  progress  assessed  against 
goals 

♦  Program  managers  encouraged  to  share 
bad  news 
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Best 

Practices 


Accountability/Rewards 

■  Sustained  leadership 

*  empower — ensure  program  managers  have 
authority  to  make  decisions  based  on 
quantifiable  data 

*  unwavering  commitment  to  trusted  program 
managers 

*  encourage  collaboration  and  communication 

■  Program  manager 

*  develop  product  knowledge 

*  meet  cost,  schedule,  performance  goals 

*  communicate  problems  and  risk 

*  implement  risk  mitigation  strategies 

■  Tenure 

■  Rewards 
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Best 

Practices 


Other  Success  Factors 

■  Disciplined,  standard  processes 

■  Lessons  learned 

★  identify  what  worked  well  and  what 
didn’t 

■  Training/mentoring 

■  Teamwork 
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DOD 

Practices 


DOD  Practices 

■  DOD  program  manager  environment 

■  DOD  layers  of  oversight 

■  Investment  strategy 

■  Business  case 

■  Knowledge-based  product 
development 

■  Accountability/rewards 

■  Incentives 

■  Formal  vs.  informal  authority 

■  Obstacles 

■  Authority 
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DOD 

Practices 


DOD  Program  Manager  Environment 


X 

n 


Senior 

leadership 


No  short-  or  long-term 

investment  strategy 


No  accountability; 
programs  are  allowed  to 
incur  significant  delays 
and  cost  increases  and  to 
be  continually  rebaselined 


Sporadic  support 
from  senior 
leadership 


CONCEPT  DEVELOPMENT 


Evolutionary 
development  adopted  in 
policy,  but  programs 
still  purse  significant 
capability  leaps 


PROGRAM 

START 


DESIGN 


PRODUCTION 


Policy  encourages  gaps  between 
resources  and  requirements  to  be 
closed,  but  programs  often  move 
forward  with  unstable  requirements 

and  technology 


Knowledge-based  process  is 
encouraged  but  not  followed;  DOD 
lacks  management  controls  to  enforce 
process;  program  managers 
incentivized  to  suppress  bad  news 


Program  manager  does  not  stay  through  execution 
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DOD 

Practices 


Oversight  Layers 


DOD 


Includes  White 
House  (OMB), 
Congress, 
Government 
Accountability  Office 


Includes  Secretary, 
Under  Secretary, 
Comptroller, 
Acquisition 
Executive,  Operating 
Command  Executive 


Includes  Secretary;  Deputy  Under  Secretary; 
Under  Secretary  for  Acquisition  Technology  & 
Logistics;  Comptroller;  Assistant  Secretary  for 
command,  Control  Communication  and 
Intelligence;  Director,  Operational  Test  & 
Evaluation;  Assistant  Secretary  (Intelligence 
Oversight;  Inspector  General;  Joint  Chiefs  of  Staff 


Includes  Defense  Contract 
Audits  Agency,  Defense 
Contract  Audit  Agency,  Defense 
Finance  and  Accounting 
Service,  Defense  Information 
Systems  Agency,  Defense 
Intelligence  Agency 


Best 

practices 


Includes  CEO,  COO,  CFO, 
Chief  Engineer,  and 
sometimes  project  office 
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DOD 

Practices 


Investment  Strategy 

■  Overarching  investment  strategy  with 
senior  leader  commitment  is  missing 

★  long-term  vision,  but  no  defined  strategy 

★  leadership — many  layers 

★  short-  and  long-term  investment 
prioritization  is  lacking  and  trade-offs 
among  programs  are  not  made 

■  DOD  starts  more  programs  than  it  can 
afford 

★  programs  compete  for  funding 
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DOD 

Practices 


Business  Case 


DOD  policies 

DOD  practices 

■  Match  requirements  to 
resources 

■  Requirements  are 
rarely  stable  or 
matched  to  resources 

■  Divide  development 
into  evolutionary 
blocks 

■  Revolutionary 
development  is  the 
norm 

■  Use  mature 
technologies 

■  Technologies  are  not 
mature 

■  Provide  full  funding 

■  Unwavering 
leadership  and  funding 
commitment  not 
provided 
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DOD 

Practices 


Knowledge-Based  Product 

Development 

■  Policy 

★  encourages  quantifiable  data  at 
milestone  decision  points 

■  Practice 

★  data  either  ignored  or  not  collected 

♦  percent  of  design  drawings 
completed  less  used 

•  percent  of  production  processes 
under  statistical  controls  not 
required 
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DOD 

Practices 


Knowledge-Based  Product 
Development  (cont’d) 


Low  levels  of  knowledge  predict  increases 
in  cost: 

Program 

Percent 
increase  in 

R&D 

Percent  of  critical 

technologies 

mature 

ATIRCMS 

5.6 

50%  (3  of  6) 

C-5  RERP 

2.1 

100%  (11  of  11) 

DD(X)  Destroyer 

417.3 

25%  (3  of  12) 

Future  Combat 
System 

50.8 

32%  (17  of  52) 

Joint  Strike  Fighter 

30.1 

25%  (2  of  8) 
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DOD 

Practices 


Knowledge-Based  Product 
Development  (cont’d) 


Knowledge-based  development  processes 
used  to  a  great  extent: 

Processes 

Program 

managers 

Program 

executive 

officers 

Technology  readiness 
levels 

32% 

23% 

Design  drawings 
complete 

32% 

11% 

Statistical  control  of 
production  processes 

26% 

14% 
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DOD 

Practices 


Accountability/Rewards 

■  Policy 

★  Program  managers  accountable  for  cost, 
schedule  and  performance 

★  Rewards — intrinsic 

■  Practice — Accountability  difficult  to  enforce 

★  Program  managers  lack  authority  over 
requirements  and  key  resources,  such  as 
funding  and  personnel. 

★  Program  managers  have  limited  tenure 

★  Practice  lies  more  in  maintaining  priority  of 
program  than  in  managing  for  outcomes 

Program  managers:  “We  can't  be  held 
accountable  for  what  we  don't  control” 
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DOD 

Practices 


Incentives 
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20 
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Improve  Financial 
career 

management 


Extend  Reduce  Increase 

tours  oversight  program 

manager 

authority 

Incentives 


Increase 

staffing 
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DOD 

Practices 


Formal  vs.  Informal  Authority 


Task 

%  formal 
authority 

%  informal 
influence 

Requirements 

10 

82 

Changes  to  requirements 

13 

85 

Technology  development 

42 

45 

Approved  program  baseline 

72 

22 

Testing  requirements 

48 

49 

Request  for  proposal 

85 

11 

Contractor  selection 

48 

33 
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DOD 

Practices 


Obstacles  to  Successful 
Program  Management 


Funding  Requirements  Staffing  Oversight 

instability  instability 


Obstacles 
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DOD 

Practices 


Additional  Authority  Wanted 


Test 

Overall 


Staff 

changes 
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Conclusions 


■  DOD  needs 

★  a  better  foundation  for  program  managers  and 
more  steadfast  support  for  them  once  it 
commits  to  programs 

★  a  long-term  investment  strategy  that  can 
mitigate  risks  by  separating  long-term  wants 
from  needs 

★  a  business  case  for  new  projects  that  assures 
they  fit  into  overall  priorities  and  have 
adequate  resources  in  terms  of  time,  money 
and  technology 

If  DOD  implements  these  things  it  will  have  gone 
a  long  way  to  removing  incentives  for  negative 
behaviors  that  have  plagued  many  DOD 
projects. 
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Recommendations 


■  Develop  a  long-term  and  short¬ 
term  investment  strategy 

■  Senior  leaders  commit  to  a 
business  case  for  each  major 
weapon  development  by 
Milestone  B 

■  Develop  and  implement  a 
process  to  instill  and  sustain 
accountability  for  successful 
program  outcomes 
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DOD 

Reponse 


■  Defense  Acquisition  Performance 
Assessment 

★  December  2005 — correct  government 
induced  instability 

■  Quadrennial  Defense  Review 

★  February  2006 — reform  business 
processes 

Whether  these  efforts  will  result  in 
improved  outcomes  depends  on 
implementation  and  discipline. 
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Q&A 
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